Systems and methods for measuring the performance of a provider of communications services

ABSTRACT

Systems and methods for measuring the performance of a provider of communications service are described. A plurality of mobile communications devices measure average power received, signal to noise ratio, voice quality, blocked and dropped calls, or any combinations of such measurements. The device may also receive a set of data identifying the location of the device. The device may transmit one or more communications signals to a server computer system with such measurements and location data. The measurements and location data may be stored in a database, and new data may be created and transmitted illustrating a subset of the measurements and location data.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.11/205,277, filed on Aug. 15, 2005, entitled “Embedded WirelessBenchmarking Systems and Methods,” the contents of which areincorporated by reference herein in its entirety.

TECHNICAL FIELD

The present invention relates generally to wireless communications, andmore specifically to the measurement of the performance of wirelesscommunications networks.

BACKGROUND

There are many options in choosing a carrier for wireless voicecommunications. Specifically, the competition for customers amongwireless carriers operating cellular networks can be intense.Maintaining high quality service can be an important consideration inattracting and maintaining customers. Poor network performance can leadto customer losses and increase churn, factors which can be costly.

Drive-test systems are a primary tool used by providers to measureperformance from a subscriber perspective. Drive-test systems, as thename implies, are tools that characterize the performance of a system bydriving around and making measurements. Such systems may rely on atechnician or other professional to drive to different coverage areasand make service-quality measurements. Such systems may employ areceiver to measure performance based on a variety of quality metrics.The receiver may be as simple as a phone or as complex as a digitalreceiver specifically created to measure such metrics as pilotpollution, missing neighbors and base-station timing errors.

A limitation of this approach is that it generally requires a technicianor other professional to drive around and make the measurements. Thisoften entails the use of expensive electronic equipment, costsassociated with the vehicle, and personnel costs for the measurementtechnician. There are some drive test solutions where the measurement isdone automatically, requiring less administration by a technician. Suchsystems are often dubbed “unattended” systems, while systems requiringprofessional, ongoing measurement are often referred to as “attended”systems. Unattended systems may also be associated with a permanentlocation.

Nonetheless, there are certain inherent limitations associated with“drive test” solutions. Because of costs, measurement is oftengeographically limited to major markets and roads, and measurements areonly taken at certain times. Also, drive test solutions are typicallydirected at a single carrier, making competitive benchmarking ofmultiple carriers a challenge. Thus, there exists a need in the art tocreate solutions that address the inherent limitations associated withdrive testing, while continuing to measure the performance of a wirelessservice from the subscriber perspective at a variety of locations.

SUMMARY

Embodiments of the invention provide systems and methods for measuringthe performance of a provider of communications service. According tosome embodiments, a plurality of mobile communications devices areconfigured to receive a range of frequencies of electromagnetic waves.In some embodiments, the range may comprise one or more signals from acommunications service provider, which may be a wireless carrieroperating a cellular system. Each signal may comprise one or moremodulated voice signals. According to various embodiments, each devicemay measure a variety of signal quality metrics, which may includeaverage power received, signal to noise ratio, voice quality, blockedand dropped calls, data network metrics, or any combinations of suchmeasurements. A device may associate a time which one or more of thevarious measurements. Each device may receive a set of data identifyingthe location of the device. This location based data may be comprised ofGPS coordinates. In some embodiments, signal quality metrics aremeasured during the regular and customary use of the device by a user.

According to various embodiments, each Device transmits a communicationssignal which includes the average power received by the range, the voicequality measurement, and the location based data. According to someembodiments, a device may also transmit information on signal to noiseratio, blocked or dropped calls, or other signal quality metrics. Inother embodiments, the additional measurements and times are included inthe communications signal. In some embodiments, the measurements mayoccur at different intervals, related to time and movement. Thecommunications signal may be transmitted via GPRS, although the signalmay also be transmitted via a variety of other means known in the art.

In some embodiments, the system is further comprised of a servercomputer system, in communication with each device. The system may beconfigured to receive the communications signals from the devices, andstore the information included in the communications signal in adatabase. In some embodiments, the system receives signal qualitymetrics representing different providers of communications services. Invarious embodiments, the system may transmit subsets of the information.The system may also use the information to create image data or tablesthat illustrate the measurements and performance in different geographicregions. In some embodiments, a method for providing a user interfacewhich illustrates performance of a provider of communications servicesis described. The methods of the present invention may also be embodiedin a computer-readable storage medium having a computer-readable programembodied therein.

Further areas of applicability of the present invention will becomeapparent from the detailed description provided hereinafter. It shouldbe understood that the detailed description and specific examples, whileindicating various embodiments of the invention, are intended forpurposes of illustration only and are not intended to necessarily limitthe scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

A further understanding of the nature and advantages of the presentinvention may be realized by reference to the following drawings. In theappended figures, similar components or features may have the samereference label. Further, various components of the same type may bedistinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

FIG. 1A illustrates a communications system that may be used to measurethe performance of a provider of communications services according tovarious embodiments of the present invention.

FIG. 1B represents an example of an interface showing a tableillustrating measurement of the performance of a provider ofcommunication services according to various embodiments of the presentinvention.

FIG. 1C represents an example of image data for measuring theperformance of a provider of communication services according to variousembodiments of the present invention.

FIG. 1D represents another example of image data for measuring theperformance of a provider of communication services according to variousembodiments of the present invention.

FIG. 1E represents an example of an interface showing a table with imagedata, illustrating measurement of the performance of a provider ofcommunication services according to various embodiments of the presentinvention.

FIG. 1F represents an example of a table illustrating certain datapoints for measuring the performance of a provider of communicationservices according to various embodiments of the present invention.

FIG. 1G represents another example of a table illustrating certain datapoints for measuring the performance of a provider of communicationservices according to various embodiments of the present invention.

FIG. 2 is flow diagram that illustrates a system for measuring theperformance of a provider of communication services according to variousembodiments of the present invention.

FIG. 3 is flow diagram that illustrates a method for measuring theperformance of a provider of communication services according to variousembodiments of the present invention.

FIG. 4 is flow diagram that illustrates a method of creating a databasefor measuring the performance of a provider of communication servicesaccording to various embodiments of the present invention.

FIG. 5 is flow diagram that illustrates the configuration of a device tobe used for measuring the performance of a provider of communicationservices according to various embodiments of the present invention.

FIG. 6 is an exemplary embodiment of a communications system formeasuring the performance of a provider of communication servicesaccording to various embodiments of the present invention.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Various embodiments of the present invention comprise systems andmethods for measuring the performance of wireless carriers. A number ofsignal quality metrics may be captured passively from actual users asthey go about their regular calling activity. Phones or othercommunication devices may be configured to make the qualitymeasurements, and transmit the collected data to a database. Thecollected data may be parsed in a variety of different ways, and may bemade available in real time over a web-based query and reporting tool.Quality metrics are therefore not limited to major markets or roads, andinstead may be collected wherever users use their handsets. Theperformance data may thus be collected in real time over real worldroutes and usage scenarios.

This description provides exemplary embodiments only, and is notintended to limit the scope, applicability or configuration of theinvention. Rather, the ensuing description of the embodiments willprovide those skilled in the art with an enabling description forimplementing embodiments of the invention. Various changes may be madein the function and arrangement of elements without departing from thespirit and scope of the invention as set forth in the appended claims.

Specific details are given in the following description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments maybe practiced without these specific details. For example, well-knowncircuits, processes, algorithms, structures, and techniques may be shownwithout unnecessary detail in order to avoid obscuring the embodiments.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart may describethe operations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be re-arranged. A process is terminated when itsoperations are completed, but could have additional steps not includedin the figure.

Moreover, as disclosed herein, the terms “storage medium” or “storagemedia” may represent one or more devices for storing data, includingread only memory (ROM), random access memory (RAM), magnetic RAM, corememory, magnetic disk storage mediums, optical storage mediums, flashmemory devices or other machine readable mediums for storinginformation. The term “computer-readable medium” includes, but is notlimited to, portable or fixed storage devices, optical storage devices,wireless channels, a sim card, other smart cards, and various othermediums capable of storing, containing or carrying instructions or data.

Furthermore, embodiments may be implemented by hardware, software,firmware, middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middlewareor microcode, the program code or code segments to perform the necessarytasks may be stored in a machine readable medium such as a storagemedium. Processors may perform the necessary tasks.

I. Overview: Embodiments of the invention provide novel solutions,including systems, methods, and software, for measuring the performanceof a provider of communications services. Certain embodiments of thepresent invention are illustrated in FIG. 1A. These embodiments may beincluded in a communications system 100, comprised of a plurality ofMobile Communications Devices 105, one or more Mobile CommunicationsBase Stations 110, Service Provider Equipment 115 from one or moreproviders of communications services, and a Server Computer System 120with an associated Database 125. A description of these elementsfollows.

The communications system components may be connected via a Network 135,which may be any combination of the following: the Internet, an IPnetwork, an intranet, a wide-area network (“WAN”), a local-area network(“LAN”), a virtual private network, the Public Switched TelephoneNetwork (“PSTN”) 130, or any other type of network supportingcommunication between devices described herein, in differentembodiments. The Network 135 may include both wired and wirelessconnections, including optical links. Many other examples are possibleand apparent to those skilled in the art in light of this disclosure. Inthe discussion that follows, the Network 135 may or may not be notedspecifically. If no specific means of connection is noted, it may beassumed that the link, communication or other connection between devicesmay be via the Network 135.

A. Device: According to different embodiments of the present invention,a Mobile Communications Device 105 may be a cellular phone, a VoIPphone, a personal digital assistant, a pager, a text messaging device, alaptop, a portable digital music player, a two way radio, any mobilephone or other device that communicates data signals, voice or otheraudio signals, or any combination of the foregoing. In variousembodiments, the Device 105 is a phone being used for voice calls by anactual user, wherein the performance metrics are being capturedpassively by the Device 105 as the user acts as a passive agent whilegoing about his or her regular calling activity. Thus, signal qualitymeasurement may occur during the regular and customary usage of a phoneor other device, and may occur while a consumer uses the phone or otherdevice for purposes of voice communication. According to someembodiments, a Mobile Communications Device 105 communicates with thepublic switched telephone network (“PSTN”) 130 through a MobileCommunications Base Station 110, wherein at least a portion of theconnection is wireless.

As used in the Application, “cellular” should be interpreted in a broadsense to include any of the variety of known modes of wireless or mobilevoice communications. Exemplary cellular systems include, but are notlimited to, time division multiple access (“TDMA”) systems, codedivision multiple access (“CDMA”) systems, and Global System for Mobilecommunications (“GSM”) systems. Other exemplary cellular systems includesystems known in the art as “3G” systems, and Enhanced Data Rates forGSM Evolution (“EDGE”) systems.

The term “VoIP” as used herein should be interpreted to mean any type ofvoice service that is provided over a data network, such as an InternetProtocol based network. The term VoIP is intended to be interpretedbroadly to include any system wherein a voice signal from a phone isconverted into a digital signal that travels over a data network. VoIPalso includes any system wherein a digital signal from a data network isdelivered to a phone or other mobile voice communications device, whereit is converted into an audio signal.

According to some embodiments, the Mobile Communications Device 105 maypreferably be comprised of hardware elements that are electricallycoupled via bus, including a microprocessor, a wireless receivercomponent, a wireless transmitter component, a computer-readable storagemedia reader, and a memory. Additionally, the storage media reader isfurther connected to a storage medium. In some embodiments, the Device105 is further comprised of a SIM card, JavaCard, or other smart card,which may include a microprocessor and storage medium. Additionalcomponents may include a microphone and speaker. There may also be othercomponents communicatively coupled to the Device 105, including anoutput component, such as a screen or other display, and an inputcomponent such as a keyboard or touchscreen. Those skilled in the artwill recognize that this is but one of a number of possibleconfigurations for the Mobile Communications Device 105.

The Mobile Communications Device 105 may also be coupled to a receivercomponent which enables the reception of a set of data identifying thelocation of the Device 105. This data may be referred to hereinafter as“location based data,” or alternatively “set of data identifying thelocation of the device.” The receiver may be coupled to the MobileCommunications Device 105 in any suitable manner known in the art. Byway of example, it may be an integrated component or may be a standalone receiver otherwise communicating with the Mobile CommunicationsDevice 105. In any case, a Mobile Communications Device 105, as the termis used herein, comprises any of the foregoing configurations.

The location based data may be in the form of Satellite locationinformation (such as Global Positioning System (“GPS”) information),cellular location information, network analysis of location information,location information specific to a building, or other means for locationdetermination. The location based data may be based on triangulationusing cellular towers or access points. Alternatively, cellular carriersmay employ other means of locating cellular telephones and other mobilecomputing devices using cellular towers. By way of example, the timedifference of arrival, angle of arrival, and location pattern matchingmethods are well known in the art as alternative means of obtaininglocation information. In some embodiments, the Device 105 receiveslocation based data. According to some embodiments, the location baseddata may be in the form of GPS coordinates. The location based data maybe stored in the Database 125, as described below.

According to various embodiments, a Mobile Communications Device 105 isconfigured to receive a range of frequencies of electromagnetic waves.The range may be received from one or more Mobile Communications BaseStations 110, or from alternative sources. According to someembodiments, the range is associated with a communications serviceprovider. The range may be comprised of one, or more, signals from awireless carrier or other wireless communications provider. The rangemay be comprised of any combination of modulated, multiplexed,digitized, packetized or otherwise encoded signals from a provider ofcommunications services. According to some embodiments, the rangecomprises at least one modulated voice signal. As used herein, modulatedvoice signal includes any audio or voice signal that is digitallyencoded.

Service providers for cellular and other systems are typically grantedor otherwise purchase ranges of frequencies for use in certaingeographic areas. However, in some cases, more than one service providermay be associated with a given range of frequencies (as may be the casefor a VoIP provider—an example of this type of shared connection is a“WiFi” hot spot that complies with the wireless standards developed inassociation with IEEE 802.11). The ranges of frequencies granted todifferent cellular service providers typically do not overlap in a givengeographic region. Individual service providers are typically grantedcontiguous blocks of frequency (often one for transmit and a differentone for receive).

A Device 105 may be coupled to any number of antennas and receivers,configured to receive the range of frequencies. A specific range offrequencies to be received may be dictated by the configuration andchoice of components of the Device 105. Alternatively, the componentsmay allow the range of frequencies to be received to be modified by theuser, the Device 105, a service provider, or another source. A Device105 may be configured to measure the average power received from therange of frequencies or portion thereof, in a manner known in the art.As noted above, the range may be comprised of one, or more, signals froma wireless carrier or other communications service provider. Thisaverage power measurement may thus comprise a signal strength, which isoften referred to as Receive Signal Strength (“RSSI”). The measurementmay be made whether a call is connected or not.

A Device 105 may be a wireless phone in regular use by a consumer ofwireless phone services. The measurement software may be loaded on avariety of commercially available mobile devices. The software may beloaded by the users themselves, and no interaction or provisioning isrequired from the carriers to be measured or the manufacturers of thedevices. Instead, any combination of the measurements may be transmittedover a data or other network and compiled in various fashions, asdescribed herein. The quality measurements described herein may becollected passively as a consumer goes about his or her normal callingactivity. The quality measurements may thus be collected in real timeover actual routes and usage patterns. In addition, the data may becollected from different service providers. A number of differenthandset design and cost considerations may govern the configuration ofthe Device 105. The design and tradeoffs associated with such handsetsand alternative devices are well known by those skilled in the art. ADevice 105 may comprise a multi-mode phone, and the components of thephone may be configured to measure more than one range of frequencies,associated with more than one service provider.

According to some embodiments, and as noted above, the range offrequencies of electromagnetic waves comprises at least one modulatedvoice signal associated with a provider of communications services. Insome embodiments, the range may comprise a plurality of modulated voicesignals. A Device 105 may be configured to measure a signal to noise(“SNR”) ratio associated with one or more of these signals, or othersignals received by the Device 105. Various methods of measuring SNR ona Device 105 are known in the art, and any of these methods may be soused in a manner consistent with this disclosure.

In some embodiments, one or more of the modulated voice signals aredemodulated to reproduce a voice or other audio signal. In such cases, aDevice 105 may be configured to measure the voice quality of thereproduced signal. The voice quality measurement functionality may bestored in the storage medium of a Device 105, or may be implemented witha combination of hardware and software. The voice quality reading maymake measurements based on the acoustical and psychological propertiesof human hearing. The Device 105 may be configured to calculate hownetwork impairments combine to affect voice quality as perceived by thecustomer. Measurements may take into account impairments such asbackground noise, echo, delay, clipping, coding errors and mismatches involume level. In VoIP there are unique sources of degradation, includingpacket loss, jitter, and latency, while cellular calls may be degradedby bit errors, frame erasures and various compression schemes. In someembodiments, the voice quality measurement comprises a Mean OpinionScore (“MOS”). The measurement may be through active or passive voicequality techniques.

Generally, the programs that provide certain functionality for theDevice 105 may be stored in the storage medium. Such software may beinstalled or updated via USB, Bluetooth, Infared, Network 135 download,or other means known in the art. Thus, in some embodiments, users mayload the software without provisioning or interaction. Instead, themeasurement may take place on a standard handset or other mobile device,and the results may be sent over a data network. Some other measurementsystems currently in use may require significant monitoring from acarrier, or special configurations from the OEM. Various embodiments ofthe invention differ, in that they comprise a software application thatcan run on many wireless networks. In light of this capability, and asdiscussed below, the results may be compiled in a database, thusallowing multi-carrier benchmarking. Different metrics for differentcarriers may be compared over a variety of times and geographic regions.

According to some embodiments, the Device is configured to query a userat the end of a call by display or other means to question whether thecall was blocked, or dropped. Such functionality may be incorporatedinto the storage medium of the device. In some embodiments, the deviceis further configured with additional functionality to measureadditional metrics related to call quality, such as latency, jitter,echo, clipping, clicking and so on. A Device 105 may also be configuredto measure the quality of a data network, measuring data speeds, SMSroundtrip time, or other metrics at various locations.

According to various embodiments, the Device 105 measures the averagepower received by the range of frequencies, the SNR associated with therange, or a combination thereof. The data from these measurements may becollected according to a variety of metrics. In some embodiments, theDevice 105 may be programmed to collect data points for thesemeasurements based on (1) the movement of the device (“distance movementmetric”), (2) the time elapsed since the last measurement (“timeinterval metric”), or (3) the earlier of the distance movement metricand time interval metric. Merely by way of example, the Device 105 maybe programmed to collect a data point if the location based dataindicates a 20 meter change in location from the previous measurement.Also, the Device 105 may be programmed to collect a data point if 10minutes elapse from the previous measurement. Further, the measurementdata points could be taken the earlier of 10 minutes, or a 20 meterchange in position. These data points may be collected all the time,whether making calls or not. Alternatively, data points may be collectedwith different algorithms according to different geographic areas,different times and days, and with different Devices 105. A time, times,or periods of time may also be associated with one or more of the datapoints or other measurements. The variety of algorithms for themeasurement and collection of such data is apparent to one skilled inthe art in light of this disclosure.

According to some embodiments, the audio, or voice, quality (hereinafter“quality measurement”) of a received voice signal is measured by aDevice 105. The quality measurement may take place during a call. Insome embodiments, the quality measurement takes place every 10 secondsduring a call, and is averaged at the end of the call. The qualitymeasurements may be taken at other intervals as well, for example, basedon different geographic areas, different times and days, and withdifferent Devices 105. In certain embodiments, the quality measurementcomprises a Mean Opinion Score (“MOS”). A time may also be associatedwith one or more of the quality measurements.

In some embodiments, a Device 105 is configured to receive informationabout a blocked or dropped call. The blocked and dropped call measuremay take place via a post call interactive questioner, wherein theprogram containing the questioner is stored in the storage medium. Theinformation may be received via other means known in the art, as well.

According to various embodiments, a Device 105 transmits informationincluding: a measure of the average power received from the range offrequencies, the SNR measure associated with the range, the qualitymeasurement, the blocked and dropped call measure, measurements of dataspeeds or other data network metrics, the times or time periods of thedata points or measures, or any combination thereof. This collection ofmetrics may be referred to, in any combination, as “signal qualitymetrics.” These transmitted signal quality metrics may be included in acommunications signal (hereinafter “communications signal”). Thecommunications signal may also include the location based data. Thecommunications signal may be transmitted at different intervals. Forexample, the intervals may be related to when each measurement takesplace, or may take place at the end of each call. Alternatively, themeasurements may be consolidated and transmitted at intervals tomaximize power preservation on a Device 105. In some embodiments, thecommunications signal is transmitted via General Packet Radio Service(“GPRS”). In other embodiments, the communications signal may comprise awireless signal to be transmitted via WiFi, WiMax, CDMA, UMTS, SMS orany other wireless means. Generally, and as described below, thecommunications signal may be directed to the Server Computer System 120and associated Database 125 (which each may be located in a Network 135)via a Mobile Communications Base Station 110. However, according todifferent embodiments, there are a variety of potential networkconfigurations and protocols to carry the communications signals.

B. Base Station: A Mobile Communications Base Station 110 is a term thatshould be construed broadly, and comprises any facility or group offacilities which sends radio signals to, or receives them from, a MobileCommunications Device 105. In some embodiments, the signals may becomprised of one or more ranges of frequencies of electromagnetic waves.The signals may be for purposes of cellular telephone communications. Insome embodiments, a Base Station 110 is comprised of a Base TransceiverStation (“BTS”) and a Base Station Controller (“BSC”). However, inalternative embodiments, a cellular Base Station 110 may be comprised ofa variety of different components known in the art, depending onperformance considerations and the specific protocols and frequencies atissue.

A Base Station 110 may also be comprised of a wireless access pointwhich sends and receives radio signals to and from the MobileCommunications Device 105. Such a wireless access point includes anydevice that provides a wireless interface. By way of example, suchaccess points may be WiFi hot spots that comply with the wirelessstandards developed in association with IEEE 802.11. Those skilled inthe art will recognize the myriad of potential configurationssurrounding a Mobile Communications Base Station 110.

A Base Station 110 may provide a communications link between a Device105 and the Server Computer System 120. The communication between aDevice 105 and the System 120 may be via a GPRS network, or any Network135. In certain embodiments, the link may be from a Device 105 through aBSC in a Base Station 110, through a Serving GPRS Support Node (“SGSN”)and a Gateway GPRS Support Node (“GGSN”) in a GPRS network, through aNetwork 135, and on to the Server Computer System 120. However, a linkbetween a Base Station 110 and the Server Computer System 120 (andassociated Database 125) may include any number of intermediate devices,including routers, switches, or other devices that receive and transmitsignals.

A Base Station 110 may also provide a communications link between aDevice 105 and Service Provider Equipment 115. Different Base Stations110 may provide the communications links between the Device 105 and theService Provider Equipment 115 of different communication serviceproviders. Different Base Stations 110 may provide the communicationslinks between the Device 105 and the Server Computer System 120 andbetween the Device 105 and the Service Provider Equipment 115. Any ofthe communications links discussed herein may include any number ofintermediate devices, including routers, switches, or other devices thatreceive and transmit signals.

C. Service Provider Equipment: As noted, the Device 105 may becommunicatively connected to Service Provider Equipment 120 from one ormore communications service providers according to various embodimentsof the invention. The Service Provider Equipment 120 may provide aninterface between the Device 105 and the PSTN 130.

The Service Provider Equipment 120 may be comprised of a mobileswitching center (“MSC”). To provide connectivity between phone users140 and the Device 105, the MSC may also be in communication with atelephone network such as the PSTN 130, and may query or otherwiseutilize a Signal Control Point (“SCP”), an additional component that iswell known to those skilled in the art.

According to other embodiments, the Base Station 110 may becommunicatively coupled or otherwise in communication with the Internet,or an other Network 135, which enables a connection to a VoIP switch(i.e. Service Provider Equipment 115). To provide connectivity betweenphone users 140, the VoIP switch may also be in communication with atelephone network such as the PSTN 130.

D. Server Computer System and Database: Various embodiments of thepresent invention include a Server Computer System 120, which mayinclude, for example, one or more suitable computing devices such asserver computers, personal computers, workstations, web servers, orother such devices. One or more of such devices that collectivelycomprise the Server Computer System 120 also comprise software elementspresent on storage media or in memory, which include an operating systemand other code.

In various embodiments, the System 120 is in communication with aplurality of Devices 105, and receives the following information: ameasure of an average power received by the range of frequencies, theSNR measure associated with the range, the quality measurement, theblocked and dropped call measure, measurements of data speeds or otherdata network metrics, the times or time periods of the data points ormeasures, or any combination thereof. As noted above, this collection ofmetrics may be referred to, in any combination, as “signal qualitymetrics.” These received signal quality metrics may be included in acommunications signal transmitted from the Devices 105. The receivedinformation may also include the location based data.

The Server Computer System 120 includes application software embodied ona computer readable medium that programs the System 120 to perform oneor more functions according to the present invention. For example,application software resident on the Server Computer System 120 may beexecutable to receive, analyze, store, or transmit measurements andlocation based data from a Device 105 via a communications signal.

According to alternative embodiments, different combinations of themeasurements may be received, analyzed, stored, or transmitted by theSystem 120. In some embodiments, the System 120 creates image dataillustrating a map, with a graphical representation of the location ofvarious received measurements from Devices 105. In some embodiments, theSystem 120 may create or transmit a table illustrating the location ofone or more Devices 105, and the average power measurements or othermeasures associated therewith. In other embodiments, the System 120determines a plurality of geographic regions linked with locations, anddisplays a table with different combinations of measurements associatedwith those locations and regions. In some embodiments, the System 120determines regions with minimum or greater signal quality metrics.Information may be retrieved from, or stored, in the Database 125.Information (such as tables, image data, or consolidated data) createdby the System 120 may be transmitted over a Network 135 to a desktopcomputer 145 or other workstation, where carriers, enterprises, or otherusers may access the information. It will be apparent to those skilledin the art that substantial variations may be implemented in accordancewith the specific requirements of the different embodiments. The ServerComputer System 120 may be fully located within a single facility ordistributed geographically, in which case a Network 135 may be used tointegrate different components of the Server Computer System 120.

The Server Computer System 120 may be associated with at least oneDatabase 125, according to various embodiments of the invention. Indifferent embodiments, the Database 125 may contain the followinginformation received from the Devices 105: a measure of an average powerreceived by the range of frequencies, the SNR measure associated withthe range, the quality measurement, the blocked and dropped callmeasure, data speeds, other data network metrics, other signal qualitymetrics, the times or time periods of the data points or measures,location based data, and geographic region data. The Database 125 mayalso contain user and Device 105 related information (e.g. MAC address,IP address, phone number, IMSI, or IMEI), and service providerinformation (e.g. service provider, system carrier, or network carrier).The Database 125 may include any number of tables and sets of tables.Application software running on the Server Computer System 120 queriesthe Database 125, and produces forms, reports, tables, images or otheroutput as dictated by the application software.

The Database 125 may be incorporated within the Server Computer System120 (e.g. within its storage media), or may be a part of a separatesystem associated with the Server Computer System 120. The Database 125may be fully located within a single facility or distributedgeographically. The Database 125 may be organized in any mannerdifferent than described above to provide the functionality called forby the various embodiments, as known by those skilled in the art.

According to various embodiments, the Database 125 includes anelectronic map database, providing data related to streets, buildings,malls, parks, lakes, rivers, mountains, and other related geographic andtopographic information. This information may be configured to becorrelated against coordinates that are produced with location baseddata received by the Server Computer System 120. Tables may be comprisedof data on large geographic areas, such as countries, states, andcounties. Tables may also be comprised of smaller geographic areas, suchas urban areas, cities, communities, and the like. Different sizes,shapes, colors, dots, icons, and fonts may be used to indicate differentfeatures. The content of the electronic map database may include anycombination of the aforementioned attributes and information. The designand organization of the aforementioned tables is discretionary andwithin the skill of those of ordinary skill in the art, given thedescriptions of data fields herein.

E. User Interface: According to various embodiments of the invention,the Database 125 may be accessed over a Network 135 via any connecteddevice 145, or via a direct connection to the Database 125. An interfacemay be used to access and parse the data, and the data may be parsed ina variety of ways. An exemplary user interface 150 which illustratesdifferent ways in which the data may be parsed is shown in FIG. 1B. Asillustrated, the data may be broken down according to the following:ranges of dates, ranges of time, subscribers, classes of subscriber,specific handset, Device 105 model, carrier, service provider,geographic region, signal quality metrics or any combinations orsubcombinations thereof. It is worth noting that data from differentcarriers and other service providers may be collected into one Database125 in real-time.

Exemplary date filter periods may include: past 24 hours, past 7 days,past month, past 3 months, past 6 months, or specific start and enddates. Exemplary time filter periods may include: past 15 minutes, past30 minutes, past hour, past 3 hours, peak AM, peak PM, off peak AM, offpeak PM, or specific start and end times. Exemplary geographic areas maybe cities, states, or specific sub-regions created in a variety of waysknown in the art. Information may be viewed along various serviceprovider or network metrics. A user interface may also illustratevarious metrics related to coverage over an area with a minimum signalquality metric. The filters and other parameters illustrated in thisparagraph are for purposes of example only, and a variety of additionalmetrics may be used as is evident to one skilled in the art.

According to some embodiments, the parsed data may be illustrated in atable, or in image data comprising a map of geographic areas ofdifferent sizes. The user interface, an example 150 of which isillustrated in FIG. 1B, may be configured to allow a user to create thetable or map according to the criteria listed above. Different colors orshades of gray may illustrate different levels of signal quality metricsfor measurements for certain geographic regions. In various embodiments,a map may illustrate major or minor roads, water, or other features. Theinterface may dictate the parameters to which the map will be drawn. Oneexample 160 of such a map for a larger region is shown in FIG. 1C.Another example 170 for a smaller region is illustrated in FIG. 1D. Theuser interface may include both a series of tables and a map, asillustrated in the example 175 shown in FIG. 1E. The user interface, andtables contained therein, may be broken down to illustrate varioussignal quality metric parameters, different carriers and providers, anddifferent subsets of subscribers, as illustrated in the example 180shown in FIG. 1F. Also, the user interface, and tables containedtherein, may be broken down to illustrate various geographic regions,and different time periods, as illustrated in the example 185 shown inFIG. 1G. With the variety of user interfaces, multi-carrier benchmarkingis possible, comparing different providers over a variety of times,dates, and geographic regions, and according to various signal qualitymetrics.

II. Different Embodiments: With a better understanding of the variouscomponents of the invention, a closer examination of specificembodiments may provide a more thorough view and illustration of thedifferent elements of the invention.

A. System: In some embodiments of the invention, a communications system200 for measuring the performance of a provider of communicationsservices is described, as illustrated in FIG. 2. This system may beimplemented within the broader communications system 100 illustrated inFIG. 1. The system 200 includes a plurality of Mobile CommunicationsDevices 105-a . . . n. Each Device 105 is configured to receive a rangeof frequencies of electromagnetic waves at block 205. The range maycomprise one or more signals from a communications service provider,which may be a wireless carrier operating a cellular system. Each Device105 may measure the average power received from the range over a periodof time, block 210, the measure comprising one or more data points. EachDevice 105 may demodulate a subset of the received range to reproduce avoice signal at block 215, and measure the voice quality of thereproduced voice signal at block 220. The voice quality measure maycomprise an MOS. At block 225, each Device 105 may receive a set of dataidentifying the location of the device. This location based data may becomprised of GPS coordinates.

At block 230, each Device transmits a communications signal whichincludes the average power received by the range, the voice qualitymeasurement, and the location based data. According to some embodiments,a Device 105 may also measure and transmit information on SNR andblocked or dropped calls. The transmission may occur at differentintervals, and one Device 105 may make multiple transmissions. Thecommunications signal may be transmitted via GPRS, although the signalmay also be transmitted via a variety of other means known in the art.

The system is further comprised of a Server Computer System 120, incommunication with each Device 105. The System 120 is configured toreceive the communications signals from all of the Devices 105 at block235, and store the information included in the communications signal ina Database 125 associated with the System at block 240.

B. Server—Method: In some embodiments of the invention, a method 300 formeasuring the performance of a provider of communications services witha server is described, as illustrated in FIG. 3. At block 305, a ServerComputer System is provided, and an example of such a Server isillustrated at 120 in FIG. 1A. At block 310, the System 120 receives atleast one communications signal from each of a plurality Devices 105. Inthis embodiment, each signal or collection of signals includes a measureof the average power received by each Device 105 for a range offrequencies of electromagnetic waves, a measurement of a signal to noiseratio for the range received, and a set of data identifying the locationof the Device 105. According to other embodiments, measurementsincluding voice quality and blocked/dropped calls may also be included.At block 315, the System 120 determines the location of each Device 105of the plurality in light of the location based data. At block 320, theSystem associates the average power and SNR measurements for each Devicewith the determined location.

At block 325, the System 120 transmits a subset of the informationrelated to the Average Power, SNR, and associated locations. Accordingto some embodiments, the subset may comprise image data containing agraphical representation of each location and the average powerassociated with each location. FIG. 1C illustrates an example 160 ofsuch image data, and FIG. 1D illustrates another such example 170. Insome embodiments, the subset comprises an interface comprising a table,an example 370 of which is illustrated is FIG. 1B.

C. Database—Method: In some embodiments of the invention, a method 300of measuring the performance of a provider of communications serviceswith a database is described, as illustrated in FIG. 4. At block 405, aServer Computer System 120 and associated database are provided. Anexample of such a database is illustrated at 125 in FIG. 1A. At block410, the System 120 receives at least one communications signal fromeach of a plurality of Devices 105. In this set of embodiments, eachsignal or collection of signals includes a measure of the average powerreceived by each Device 105 for a range of frequencies ofelectromagnetic waves. In various embodiments, the range may comprise awireless signal or signals from a wireless carrier operating a cellularnetwork. In this set of embodiments, the communications signal or set ofsignals may include measurement of the voice quality, which may comprisean MOS. The communications signal or set of signals may further includea set of data identifying the location of the Device 105, which maycomprise GPS coordinates. According to other embodiments, measurementsincluding SNR and blocked/dropped calls may also be included. In stillother embodiments, the times or time periods of the data points ormeasures may be received as well.

At block 415, the System 120 stores the information included in eachsignal or signals in the Database 125. At block 420, the Database ismaintained with the stored information. Such maintenance may include thecreation and maintenance of relational tables. By way of example, linksbetween locations and the associated average power and voice qualitymeasurements may be included. In addition, links between the locationsand SNR measurements, blocked/dropped calls, and times or time periodsof the data points or measures may be included as well.

D. Device: In some embodiments of the invention, a device 500 configuredto transmit information related to measuring the performance of aprovider of communications services is described, as illustrated in FIG.5. An example of such a device is illustrated at 105 in FIG. 1A. TheDevice 105 is configured to receive a range of frequencies ofelectromagnetic waves at block 505. The range may comprise one or moresignals from a communications service provider, which may be a wirelesscarrier operating a cellular system. A Device 105 may measure (at block520) the average power received from the range when the device has moveda sufficient distance 510, or when a sufficient time has passed 515,whichever occurs first. According to some embodiments, the distancemetric is 20 meters, and the time interval metric is 10 minutes.

Each Device 105 may demodulate a subset of the received range toreproduce a voice signal at block 525, and measure the voice quality ofthe reproduced voice signal at block 530. The voice quality measure maycomprise an MOS. At block 535, each Device 105 may receive a set of dataidentifying the location of the device. This location based data may becomprised of GPS coordinates. At block 540, each Device transmits acommunications signal which includes the average power received by therange, the voice quality measurement; and the location based data.

E. Specific Embodiment: Having shown many broad embodiments of theinvention, it may be useful to illustrate a more specific embodiment.The following embodiment is for exemplary purposes only. In thisembodiment, illustrated in FIG. 6, a phone call is connected from atelephone 140 to a Device 105. A voice signal is modulated andtransmitted 605 from the telephone 140 to the PSTN 130. From the PSTN130, the signal is forwarded 610 to an MSC (i.e. Service ProviderEquipment 115) associated with a service provider. The signal is thenforwarded 615 through a BSC and BTS (i.e. Base Station 110). Themodulated voice signal is then forwarded 620 to the Device 105 from theBTS in the form of a range of frequencies of electromagnetic waves.

The Device 105 may measure the average power received for the range offrequencies, the SNR, and the voice quality, and may also receivelocation data. The Device may then packetize and transmit 625 themeasurements and location data using GPRS, through a BTS and BSC (BaseStation 110). The GPRS packets travel 630 through an SGSN and are thenforwarded 635 through a GGSN. The packets are then transmitted 640 overthe Internet 135, to 645 the Server Computer System 120. The informationmay then be stored 650 in the Database 125. Similar transmissions aremade from a plurality of Devices 105 to measure the performance of awireless carrier.

It should be noted that the methods, systems and devices discussed aboveare intended merely to be exemplary in nature. Consequently, variousembodiments may omit, substitute, or add various procedures orcomponents as appropriate. For instance, it should be appreciated thatin alternative embodiments, the methods may be performed in an orderdifferent than that described, and that various steps may be added,omitted or combined. Also, features described with respect to certainembodiments may be combined in various other embodiments. Differentaspects and elements of the above described embodiments may be combinedin a similar manner.

Having described several embodiments, it will be recognized by those ofskill in the art that various modifications, alternative constructions,and equivalents may be used without departing from the spirit of theinvention. Accordingly, the above description should not be taken aslimiting the scope of the invention, which is defined in the followingclaims.

1. A system for measuring performance of a provider of communicationsservices, the system comprising: a server; a data storage systemconnected to the server; means for instructing each of a plurality ofmobile devices to measure an average power received from a range offrequencies at predetermined time intervals and a time at which eachmeasurement is made; means instructing each of the mobile devices todemodulate a subset of the range of frequencies to produce a demodulatedvoice signal; means for instructing the measurement of a voice qualitymeasurement of the demodulated voice signal when the mobile device is inuse, the voice quality measurement based on acoustical properties ofhuman hearing; means for identifying the location of the device; meansfor instructing the mobile communication device to collect data pointsfor measurements of the average power received, measurements of the timeat which the average power received is measured, the voice qualitymeasure and for the location of the device, wherein the data points arecollected based on a metric selected from the group consisting of a timeinterval metric and a distance movement metric; means for receiving atthe server sets of data comprising; all data points collected for themeasurement of the average power received from the range and the time atwhich the data points for the measurement of the average power receivedwere made; all data points collected for the measurement of the voicequality measurement and the time at which the data points for themeasurement of the voice quality were made; and the location of thedevice at the time at which the collection of the data points for themeasurement of the voice quality and the measurement of the averagepower received were collected; and a database in the storage system forstoring the sets of data.
 2. The system of claim 1, wherein the providerof communications services is a wireless carrier operating a cellularsystem.
 3. The system of claim 1, wherein the range of frequencies ofelectromagnetic waves received comprises a wireless signal from theprovider of communications services, and wherein the metric is adistance movement metric for determining whether the device has moved apredetermined distance.
 4. The system of claim 1, wherein the range offrequencies of electromagnetic waves received comprises a modulatedvoice signal, and wherein the metric is a time interval metric fordetermining whether a predetermined amount of time has passed since aprevious measurement.
 5. The system of claim 1, wherein the databasealso contains user and mobile device information.
 6. The system of claim1 wherein the database further comprises an electronic map database. 7.The system of claim 1 further comprising a user interface to access thedatabase.
 8. A method for measuring performance of a provider ofcommunications services, the method comprising: instructing each of aplurality of mobile devices to measure an average power received from arange of frequencies at predetermined time intervals and a time at whicheach measurement is made; instructing each of the mobile devices todemodulate a subset of the range of frequencies to produce a demodulatedvoice signal; instructing the measurement of a voice quality measurementof the demodulated voice signal when the mobile device is in use, thevoice quality measurement based on acoustical properties of humanhearing; identifying the location of the device; instructing the mobilecommunication device to collect data points for measurements of theaverage power received, measurements of the time at which the averagepower received is measured, the voice quality measure and for thelocation of the device, wherein the data points are collected based on ametric selected from the group consisting of a time interval metric anda distance movement metric; receiving at a server sets of datacomprising; all data points collected for the measurement of the averagepower received from the range and the time at which the data points forthe measurement of the average power received were made; all data pointscollected for the measurement of the voice quality measurement and thetime at which the data points for the measurement of the voice qualitywere made; and the location of the device at the time at which thecollection of the data points for the measurement of the voice qualityand the measurement of the average power received were collected; andstoring the sets of data in a database in a storage system.
 9. Themethod of claim 8, wherein the provider of communications services is awireless carrier operating a cellular system.
 10. The method of claim 8,wherein the range of frequencies of electromagnetic waves receivedcomprises a wireless signal from the provider of communicationsservices, and wherein the metric is a distance movement metric fordetermining whether the device has moved a predetermined distance. 11.The method of claim 8, wherein the range of frequencies ofelectromagnetic waves received comprises a modulated voice signal, andwherein the metric is a time interval metric for determining whether apredetermined amount of time has passed since a previous measurement.12. The method of claim 8, wherein the database also contains user andmobile device information.
 13. The method of claim 8 wherein thedatabase further comprises an electronic map database.
 14. The method ofclaim 8 further comprising a providing user interface to access thedatabase.